Method and device for detecting drivable region of road

ABSTRACT

A method and a device are disclosed for detecting a drivable region of a road, the method comprising the steps of: deriving a disparity map from a gray-scale map including the road and detecting the road from the disparity map; removing a part with a height above the road greater than a predetermined height threshold from the disparity map so as to generate a sub-disparity map; converting the sub-disparity map into a U-disparity map; detecting the drivable region from the U-disparity map; and converting the drivable region detected from the U-disparity map into the drivable region within the gray-scale map.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to road detection, andspecifically, a method and a device for detecting a drivable region of aroad.

2. Description of the Related Art

Currently, a driving assistance technology of an automobile is studied.This technology can provide needed information and/or warning to a userwhen the user drives the car, so that a dangerous situation such as avehicle collision or vehicle deviation from the road can be avoided.Furthermore, the driving assistance technology may be used to controlthe moving of the car automatically in some cases.

In the driving assistance technology, it is important to detect thedrivable region of the road correctly. Many methods for detecting thedrivable region of the road have been disclosed. For example, U.S.Patent Application Publication US20100250064A1 discloses a method fordetecting a drivable region by using the white lines on both sides ofthe road, however such a method cannot be applied to a road where thewhite lines do not exist on both sides thereof. The U.S. PatentApplication Publication US20050015201A1 discloses a method forsegmenting a road and detecting a drivable region by a distance and asegmental slope, however such a method cannot be applied to a sparsedisparity map where points on the road are sparse.

Therefore, there is a demand for providing a method and device forreliably detecting a drivable region of a road.

SUMMARY OF THE INVENTION

The present invention is made in light of the above problems. It is anobject of the present invention to provide a method and device forreliably detecting a drivable region of a road.

According to an aspect of the present invention, a method for detectinga drivable region of a road comprises the steps of: deriving a disparitymap from a gray-scale map including the road and detecting the road fromthe disparity map; removing a part with a height above the road greaterthan a predetermined height threshold from the disparity map so as togenerate a sub-disparity map; converting the sub-disparity map into aU-disparity map; detecting the drivable region from the U-disparity map;and converting the drivable region detected from the U-disparity mapinto the drivable region within the gray-scale map.

According to the above aspect of the present invention, the step ofdetecting the road from the disparity map may comprise the steps of:converting the disparity map into a V-disparity map; and fitting a roadline representing the road in the V-disparity map.

According to the above aspect of the present invention, thesub-disparity map may be generated by removing a part that is locatedabove the road line and has a distance from the road line greater thanthe predetermined height threshold from the disparity map.

According to the above aspect of the present invention, the step ofdetecting the drivable region from the U-disparity map may comprise thesteps of: selecting an initial position within the drivable region fromthe U-disparity map; and detecting a specific region from theU-disparity map as the drivable region by shifting a detection windowwith a predetermined size from the initial position in the U-disparitymap, wherein the specific region includes the initial position, and thenumber of active points with a brightness greater than a brightnessthreshold within the detection window is less than a predeterminednumber threshold during the time when the detection window is shifted inthe specific region.

According to the above aspect of the present invention, the step ofdetecting a specific region from the U-disparity map as the drivableregion by shifting a detection window with a predetermined size from theinitial position in the U-disparity map may comprise the steps of: (a)horizontally shifting the detection window along a direction of rows tothe left from the initial position, and stopping the shifting of thedetection window and recording the stop position of the detection windowas a left stop position of the said row if a first determined conditionis met, horizontally shifting the detection window along a direction ofrows to the right from the initial position, and stopping the shiftingof the detection window and recording the stop position of the detectionwindow as a right stop position of the said row if a second determinedcondition is met; (b) shifting the detection window upward by a distanceequal to the height of the detection window, and repeating step (a); (c)repeating step (b) until the detection window reaches a top edge of theU-disparity map or the left stop position and the right stop position inthe same row are adjacent to each other; and (d) extracting a regioncovered by the shifted detection windows from the U-disparity map as thedrivable region, wherein the first predetermined condition is one of thecondition that the number of the active points within the detectionwindow is greater than a predetermined number threshold; the conditionthat the detection window reaches a left edge of the U-disparity map,even though the number of the active points within the detection windowis not greater than the predetermined number threshold; and thecondition that the horizontal distance between the position of thedetection window and the left stop position of the previous row is lessthan a first distance threshold, even though the detection window doesnot reach the left edge of the U-disparity map and the number of theactive points within the detection window is not greater than thepredetermined number threshold, wherein the second predeterminedcondition is one of the condition that the number of the active pointswithin the detection window is greater than a predetermined numberthreshold; the condition that the detection window reaches a right edgeof the U-disparity map, even though the number of the active pointswithin the detection window is not greater than the predetermined numberthreshold; and the condition that the horizontal distance between theposition of the detection window and the right stop position of theprevious row is less than a second distance threshold, even though thedetection window does not reach the right edge of the U-disparity mapand the number of the active points within the detection window is notgreater than the predetermined number threshold.

According to the above aspect of the present invention, the step ofconverting the drivable region detected from the U-disparity map intothe drivable region within the gray-scale map may comprise the steps of:obtaining coordinates of points within the drivable region detected fromthe U-disparity map; calculating coordinates of points within thedrivable region of the gray-scale map based on the coordinates of thepoints, by an expression for fitting the road line in the V-disparitymap; and extracting the drivable region within the gray-scale map basedon the coordinates of the points within the drivable region of thegray-scale map.

According to another aspect of the present invention, a device fordetecting a drivable region of a road comprise: a road detectingapparatus configured to derive a disparity map from a gray-scale mapincluding the road and detect the road from the disparity map; asub-disparity map generating apparatus configured to remove a part witha height above the road greater than a predetermined height thresholdfrom the disparity map so as to generate a sub-disparity map; aU-disparity map generating apparatus configured to convert thesub-disparity map into a U-disparity map; a U-disparity map detectingapparatus configured to detect the drivable region from the U-disparitymap; and a drivable region converting apparatus configured to convertthe drivable region detected from the U-disparity map into the drivableregion within the gray-scale map.

According to the above aspect of the present invention, the roaddetecting apparatus may comprise: a V-disparity map generating unitconfigured to convert the disparity map into a V-disparity map; and aroad line fitting unit configured to fit a road line representing theroad in the V-disparity map.

According to the above aspect of the present invention, the U-disparitymap detecting apparatus may comprise: a selection unit configured toselect an initial position within the drivable region from theU-disparity map; and a detection unit configured to detect a specificregion from the U-disparity map as the drivable region by shifting adetection window with a predetermined size from the initial position inthe U-disparity map, wherein the specific region includes the initialposition, and the number of active points with a brightness greater thana brightness threshold within the detection window is less than apredetermined number threshold during the time when the detection windowis shifted in the specific region.

According to the above aspect of the present invention, the drivableregion converting apparatus may comprise: a coordinate extracting unitconfigured to obtain coordinates of points within the drivable regiondetected from the U-disparity map; a conversion unit configured tocalculate coordinates of points within the drivable region of thegray-scale map based on the coordinates of the points, by an expressionfor fitting the road line in the V-disparity map; and a drivable regionextracting unit configured to extract the drivable region within thegray-scale map based on the coordinates of the points within thedrivable region of the gray-scale map.

The method and the device for detecting the drivable region of the roadaccording to the above aspect of the present invention have a wideapplication range, because they do not depend on white lines or a fenceon the road. Additionally, the method and the device detect the drivableregion based on the density of the active points within the detectionwindow, according to the U-disparity map obtained from the disparitymap; therefore, they have high anti-noise robustness and reliability.

BRIEF DESCRIPTION OF THE DRAWINGS

The above object and other objects of the present invention will becomemore apparent from the following detailed description of the embodimentsof the present invention when read in conjunction with the accompanyingdrawings.

FIG. 1 is a schematic drawing illustrating a process of deriving aV-disparity map from a disparity map;

FIG. 2 is a schematic drawing illustrating a process of deriving aU-disparity map from a disparity map;

FIG. 3 is a schematic drawing illustrating a system capable of executingthe method for detecting a drivable region of a road according to theembodiment of the present invention;

FIG. 4 is a flowchart illustrating the method for detecting the drivableregion of the road according to the embodiment of the present invention;

FIG. 5 is a schematic drawing illustrating an example of one of the leftand right images picked up by a binocular camera;

FIG. 6 is a schematic drawing illustrating a disparity map calculated byusing the image illustrated in FIG. 5 as a reference image;

FIG. 7 is a schematic drawing illustrating a V-disparity map obtained byconverting the disparity map illustrated in FIG. 6;

FIG. 8 is a schematic drawing illustrating a road line obtained byperforming a fitting in the V-disparity map illustrated in FIG. 7;

FIG. 9 is a schematic drawing illustrating the calculated distancebetween a point that is located above the road and the road in thedisparity map;

FIG. 10 is a schematic drawing illustrating a sub-disparity map obtainedby removing a part with a height above the road greater than apredetermined height threshold from the disparity map illustrated inFIG. 6;

FIG. 11 is a schematic drawing illustrating a U-disparity map obtainedby converting the sub-disparity map illustrated in FIG. 10;

FIG. 12 is a schematic drawing illustrating a process of selecting aninitial position from the U-disparity map illustrated in FIG. 11 andsetting a detection window;

FIG. 13 is a schematic drawing illustrating a process of shifting thedetection window from the initial position to the left;

FIG. 14 is a schematic drawing illustrating the determined left stopposition of the detection window in said row;

FIG. 15 is a schematic drawing illustrating a process of shifting thedetection window from the initial position to the right;

FIG. 16 is a schematic drawing illustrating the determined right stopposition of the detection window in said row;

FIG. 17 is a schematic drawing illustrating the initial position aftershifting the detection window upward;

FIG. 18 is a schematic drawing illustrating a process of shifting thedetection window from the initial position illustrated in FIG. 17 to theleft;

FIG. 19 is a schematic drawing illustrating the determined left stopposition of the detection window in said row;

FIG. 20 is a schematic drawing illustrating a process of shifting thedetection window from the initial position illustrated in FIG. 17 to theright;

FIG. 21 is a schematic drawing illustrating the determined right stopposition of the detection window in said row;

FIG. 22 is a schematic drawing illustrating the drivable region detectedfrom the U-disparity map illustrated in FIG. 11;

FIG. 23 is a schematic drawing illustrating a coordinate conversionbetween the U-disparity map and a gray-scale map;

FIG. 24 is a schematic drawing illustrating the drivable region of theroad detected from the gray-scale map illustrated in FIG. 5;

FIG. 25 is a block diagram illustrating a device for detecting thedrivable region of the road according to the embodiment of the presentinvention;

FIG. 26 is a block diagram illustrating the detailed structure of a roaddetecting apparatus illustrated in FIG. 25;

FIG. 27 is a block diagram illustrating the detailed structure of aU-disparity map detecting apparatus illustrated in FIG. 25;

FIG. 28 is a block diagram illustrating the detailed structure of adrivable region converting apparatus illustrated in FIG. 25;

FIG. 29 is schematic drawing illustrating the hardware configuration forimplementing the device for detecting the drivable region of the roadaccording to the embodiment of the present invention.

It should be noted that the above drawings may be not drawn to scale,such drawings are for descriptive purposes only and the presentinvention is not limited to such drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following, embodiments of the present invention are describedwith reference to the accompanying drawings, so as to facilitate theunderstanding of the present invention.

Introduction of Essential Concepts

In the embodiments of the present invention, concepts of “disparity”,“disparity map”, “V-disparity map” and “U-disparity map” are used. Theseconcepts are well-known in the art, therefore these concepts will bebriefly introduced as follows.

“Disparity” indicates an angle made by two straight lines that are drawnfrom opposite ends of a base line to a distant object. In general, itmeans a direction difference generated by looking at the same objectfrom two points having a certain separation distance. The angle that isgenerated by looking at the two points from the object is called a“disparity angle” of the two points, and the distance between the twopoints is called a “base line”. A distance between the object and anobserver can be calculated as long as the disparity angle and length ofthe base line are known.

“Disparity map” is an image that is based on a reference image, the sizeof the disparity map is the size of the reference image, points (pixels)of the disparity map correspond to points of the reference image thathave the same coordinates, and gray scales of the points are images ofthe disparities of the corresponding points within the reference image.The disparity map may be obtained by a variety methods in the art. Forexample, the disparity map may be calculated from left and right imagespicked up by a binocular camera, be calculated from images picked up bya multi-camera or a stereo camera, or be calculated from a depth map ofa stereogram. In the case where left and right images are picked up by abinocular camera, a SAD (Sum of Absolute Differences) algorithm or otheralgorithms in the art may be used on the left and right images tocalculate the disparity map.

The coordinates of a point of a reference image may be represented as(x,y), where x is abscissa and y is ordinate, and the smaller y is, thenearer the point from the camera. In the two-dimensional disparity mapobtained by converting the reference image, the coordinates of a pointcorresponding to the point (x,y) may be represented as (u,v), where u isabscissa, v is ordinate, u=x and v=y. The gray value of each of pointswithin the disparity map is the disparity of the point, and isrepresented as d. Alternatively, the point within the disparity map maybe represented as (u,v,d) so as to reflect the coordinates and thedisparity of the point simultaneously.

The V-disparity map and the U-disparity map may be derived from thedisparity map. The V-disparity may be regarded as a side view of thedisparity map, where the abscissa axis is the d axis, the ordinate axisis the v axis, the point may be represented as (d,v), and the gray value(brightness value) of the point (d,v) is the number of the points of thecorresponding disparity map where the ordinate is v and the disparityvalue is equal to d. Therefore, the V-disparity map may be derived fromthe disparity map by the statistics of the numbers or other methods inthe art. FIG. 1 is a schematic drawing illustrating a process ofderiving a V-disparity map from a disparity map. The U-disparity may beregarded as a top view of the disparity map, where the abscissa axis isthe u axis, the ordinate axis is the d axis, the point may berepresented as (u,d), and the gray value (brightness value) of the point(u,d) is the number of the points of the corresponding disparity mapwhere the abscissa is u and the disparity value is equal to d.Therefore, the U-disparity map may be derived from the disparity map bythe statistics of the numbers or other methods in the art. FIG. 2 is aschematic drawing illustrating a process of deriving a U-disparity mapfrom a disparity map.

EMBODIMENT

In the following, the method and the device for detecting the drivableregion of the road according to the embodiment of the present inventionwill be described with reference to the accompanying drawings.

First, the method for detecting the drivable region of the roadaccording to the embodiment of the present invention will be described.Such method may be performed by, for example, the system illustrated inFIG. 3. As illustrated in FIG. 3, a binocular camera may be mounted in acar. When the car moves on the road, left and right images in front ofthe car that include the road (road surface) are picked up by thebinocular camera. The images are input into a chip for processing theleft and right images so as to obtain the disparity map, and thedrivable region on the road is detected from the disparity map accordingto the following method as described below.

FIG. 4 is a flowchart illustrating the method for detecting the drivableregion of the road according to the embodiment of the present invention.

As illustrated in FIG. 4, in step S401, a disparity map is derived froma gray-scale map including the road, and the road is detected from thedisparity map.

In the case where a binocular camera is used, left and right images(gray scale images) including the road may be previously obtained. Thebinocular camera may be mounted in the car, and left and right grayscale images in front of the car that include the road are picked up; orleft and right color images including the road may be picked up by thebinocular camera, and be converted into gray scale images. FIG. 5 is aschematic drawing illustrating an example of one of the left and rightimages picked up by a binocular camera. In the following, such an imageis described as an example. As illustrated in FIG. 5, a coordinatesystem is built into such images, and points thereof may be representedas (x, y).

Next, any one image from the left and right images may be selected asthe reference image, and the disparity map including the road is derivedfrom the left and right images. The disparity map may be calculated fromthe left and right images by the above method, description of this isomitted here. FIG. 6 is a schematic drawing illustrating a disparity mapcalculated by using the image illustrated in FIG. 5 as a referenceimage. As illustrated in FIG. 6, a U-V coordinate system is built intothe disparity map, and the point within the disparity map correspondingto the point (x, y) may be represented as (u, v, d), where u=x, v=y, dis the disparity of the point. It should be noted that the edges of anobject (such as the edge of the road, an obstacle on the road and theedge of objects except the road) are easily identified, and it isdifficult to identify the parts of the road that do not have an edge,when the disparity map is calculated from the left and right images by amatching algorithm; therefore, in the disparity map, the positionscorresponding to the edges have dense points, and the parts without anedge such as the drivable region of the road have sparse points.

In the following, the road is detected based on the disparity map.

Specifically, the disparity map including the road may be converted intoa V-disparity map. Such conversion may be performed by the method asdescribed above, description of this is omitted here. FIG. 7 is aschematic drawing illustrating a V-disparity map obtained by convertingthe disparity map illustrated in FIG. 6.

Next, a fitting of a road line representing the road may be performed inthe V-disparity map. In the image picked up by the camera mounted on thecar, the road extends upward and becomes narrower from the bottom of theimage. In the top side of the image, other non-road objects increase.Accordingly, in the V-disparity map, most of the points corresponding tothe road are located in the lower part of all the points that areillustrated in the drawing, and most of the points of the upper partcorrespond to other non-road objects. Therefore, the lowest part of allthe points illustrated in the V-disparity map may be selected to performthe fitting, so that the road line representing the road can be fit andan expression of the road line v=kd+b can be obtained, where k is theslope of the road line and b is a constant. For example, the fitting maybe performed by a method proposed in the paper “Detection of Non-flatGround Surfaces Using V-Disparity Images” written by Jun Zhao, MarkAlbert Whitty and Jayantha Katupitiya, 2009 IEEE/RSJ InternationalConference on Intelligent Robots and Systems, Oct. 11-15, 2009, St.Louis, USA. In the example illustrated in FIG. 7, a road lineillustrated in FIG. 8 that the expression thereof is v=0.5d+233 can beobtained by selecting the lowest part of all the points according to theabove method and performing the fitting.

Returning to FIG. 4, in step S402, a part with a height above the roadgreater than a predetermined height threshold is removed from thedisparity map so as to generate a sub-disparity map.

It is generally known that there may be tall objects such as telephonepoles, trees and columns on both sides of the road. Additionally, theremay be overhead wires above the road. In the detection of the drivableregion of the road by the U-disparity map as described below, thesenon-road objects may be reflected in the U-disparity map and detectedincorrectly as the road, if proper processing for these objects is notperformed.

In order to avoid such a problem, considering the fact that the aboveobjects have a certain height from the road surface, a height thresholdmay be predetermined, the part with a height above the road greater thanthe predetermined height threshold is removed from the disparity map soas to generate the sub-disparity map, and the U-disparity map isgenerated by the sub-disparity map rather than the disparity map. Asdescribed above, the road line within the V-disparity map represents theroad, therefore the sub-disparity map may be generated by removing apart that is located above the road line and has a distance from theroad line greater than the predetermined height threshold from thedisparity map. Such a height threshold may be freely set as, forexample, 4 meters, based on the actual situation of the road (such asthe type of the obstacle, the height and the distribution).

Specifically, first, the height from the road of each of the pointsabove the road in the disparity map p(u′, v′, d′) may be calculated. Thecoordinate of point p(u′, v′, d′) within the V-disparity map is (d′,v′). The expression of the road line within the V-disparity map isv=kd+b; therefore, the coordinate of point r with a disparity d′ that islocated above the road line (the road) in the V-disparity map is (d′,kd′+b). In the V-disparity map, the pixel distance between the point pand the point r (or the road) is v′−(kd′+b), that is, in the V-disparitymap, the distance between the point p and the road is v′−(kd′+b) pixels.The height of the point p to the road within the physical space may becalculated by multiplying such pixel distance by an actual distancewithin the physical space corresponding to each pixel. The actualdistance within the physical space corresponding to each pixel m may becalculated by various methods in the art. For example, the actualdistance m may be calculated by the method proposed in “LearningOpenCV”, O'Reilly Media, written by Gary Bradski and Adrian Kaebler, thedescription of this method is omitted here. In the example illustratedin FIG. 8, with respect to the point (400,800,600) within the disparitymap, the pixel distance thereof to the road may be calculated as800−(0.5*600+233)=267, according to the above method. Supposing theactual distance corresponding to each pixel m is 4 mm, the height of thepoint to the road is 267 pixels*4 mm/pixel=1068 mm, as illustrated inFIG. 9.

Next, a comparison between the height of the point above the road withinthe disparity map and the height threshold may be performed. The pointis removed from the disparity map if the height of the point to the roadis greater than the height threshold, otherwise the point is retained.In the example illustrated in FIG. 9, 1068 mm is less than the heightthreshold 4 meter, therefore the point is retained.

All of the points with a height to the road greater than the heightthreshold can be determined by performing the above operation for all ofthe points above the road line within the V-disparity map, and thesub-disparity map can be generated by removing these points from thedisparity map. In the example illustrated in FIG. 8, a sub-disparity mapillustrated in FIG. 10 can be obtained, when the predetermined height isset as 4 meters.

Next, in step S403, the sub-disparity map is converted into theU-disparity map. Such conversion may be performed by the above method orother methods in the art. for example, the conversion may be performedby the method proposed in the paper “A Complete UV-Disparity Study forStereovision Based 3D Driving Environment Analysis” written by ZhenchengHu, Francisco Lamosa and Keiichi Uchimura, Proceedings of the FifthInternational Conference on 3-D Digital Imaging and Modeling 2005(3DIM′05), pp. 204-211, the detailed description is omitted here. In theabove example, the U-disparity map illustrated in FIG. 11 is obtained byconverting the sub-disparity map illustrated in FIG. 10.

Returning to FIG. 4, in step S404, the drivable region is detected fromthe U-disparity map.

As described above, in the disparity map, the positions corresponding tovarious edges (such as edges of the road and edges of obstacles on theroad) have dense points, and the part where there are no edges or asmall edge, such as the drivable region of the road have sparse points;therefore, in the U-disparity map, the points corresponding to suchedges (corresponding to a non-drivable region) have a higher brightness,and the points corresponding to the drivable region of the road have alower brightness, when such a disparity map is converted into theU-disparity map. The drivable region may be detected from theU-disparity map according to this fact. Specifically, a detection windowwith a predetermined size and a brightness threshold may be set, thedetection window is shifted from an initial position that is located inthe drivable region and is preliminarily selected from the U-disparitymap, and it is considered that the current position of the detectionwindow is located in the edge of the drivable region or out of thedrivable region of the road if the number (it may also be called a“density of active points”) of active points (points with a brightnessgreater than the brightness threshold) within the detection windowexceeds a predetermined number threshold. The size of the detectionwindow and the brightness threshold may be freely set according to anactual condition of the road or the request of the detection accuracy.Finally, a specific region can be detected from the U-disparity map asthe drivable region by shifting the detection window. The specificregion includes the initial position, and the number of active pointswith a brightness within the detection window is less than apredetermined number threshold during the time when the detection windowis shifted in the specific region. In fact, the shifting process is aprocess of gradually extending a small drivable region included in thedetection window located in the initial position so as to obtain a finaldrivable region. The initial position may be freely set as long as itlocated in the drivable region. Usually, because the vehicle is movingon the road, there is not an obstacle in front of the vehicle and such aposition is located in the drivable region; therefore, preferably, theposition corresponding to such a position, namely, the middle positionof the lowest part of the U-disparity map may be selected as the initialposition from the U-disparity map, when the drivable region is detectedbased on the gray scale image picked up by a camera mounted on thevehicle as described above.

The detection window may be shifted by various methods. Preferably, thedetection window may be shifted by an 8 adjacent areas (8-CCA) method todetect the specific region. In the following, an example of the processof shifting the detection window by the 8-CCA method will be describedwith reference to the U-disparity map illustrated in FIG. 11.

First, the initial position is selected from the U-disparity map. Asillustrated in FIG. 12, in this case, the middle position of the lowestedge within the U-disparity map is selected as the initial position. Thedetection window is placed at the initial position illustrated as block“1” in FIG. 12.

Next, the detection window is horizontally shifted along a direction ofrows to the left from the initial position as illustrated in FIG. 13,the shifting of the detection window is stopped and the stop position ofthe detection window is recorded as a left stop position of the said rowif a first determined condition is met. The first predeterminedcondition is one of the conditions: (1) the number of the active pointswithin the detection window is greater than a predetermined numberthreshold; (2) the detection window reaches a left edge of theU-disparity map, even though the number of the active points within thedetection window is not greater than the predetermined number threshold;and (3) the condition that the horizontal distance between the positionof the detection window and the left stop position of the previous rowis less than a first distance threshold, even though the detectionwindow does not reach the left edge of the U-disparity map and thenumber of the active points within the detection window is not greaterthan the predetermined number threshold. If the condition (1) is met, itis indicated that the detection window has reached or gone out of theedge of the drivable region, therefore the moving of the detectionwindow to the left should be stopped. If the condition (2) is met, it isindicated that the detection window has reached the left edge of theU-disparity map, even though the detection window is still located inthe drivable region at this time; therefore, the moving of the detectionwindow to the left should be stopped. The setting of the condition (3)is based on the following fact: in the image picked up by the camera,the closer to the lower end of the image, the wider the illustrated roadis, accordingly, the wider the illustrated drivable region is; thecloser to the upper end of the image, the narrower the illustrated roadis, accordingly, the narrower the illustrated drivable region is;therefore, the more the detection window is moved in an upper direction,the closer to a central line of the image in the horizontal directionthe left stop position thereof is. That is, the left stop position inthe current row should be located in the right side of the left stopposition of the previous row. If the condition (3) is met, it isindicated that the current position of the detection window has reacheda position near the left stop position of the previous row, even thoughthe above conditions (1) and (2) are not met; therefore, in order toavoid the detection result contravening the above fact, the moving ofthe detection window to the left should be stopped. The first distancethreshold may be freely set according to the actual condition of theroad or the accuracy requirement of the detection result. In the exampleillustrated in FIG. 13, the detection window is located in the lowestrow of the U-disparity map and the number of the active points withinthe U-disparity map is not greater than the number threshold during thetime; therefore, the detection window is shifted to the left edge of theU-disparity map, namely, the left stop position of such row is the leftedge of the U-disparity map, as illustrated in FIG. 14.

Next, the detection window is horizontally shifted along the directionof rows to the right from the initial position as illustrated in FIG.15, the shifting of the detection window is stopped and the stopposition of the detection window is recorded as a right stop position ofthe said row if a second determined condition is met. Similarly to thefirst predetermined condition, the second predetermined condition is oneof the conditions: the number of the active points within the detectionwindow is greater than a predetermined number threshold; the conditionthat the detection window reaches a right edge of the U-disparity map,even though the number of the active points within the detection windowis not greater than the predetermined number threshold; and thehorizontal distance between the position of the detection window and theright stop position of the previous row is less than a second distancethreshold, even though the detection window does not reach the rightedge of the U-disparity map and the number of the active points withinthe detection window is not greater than the predetermined numberthreshold. The first and second predetermined conditions may be the sameor different, but the first and second predetermined conditions arepreferably set as the same. In the example illustrated in FIG. 15,similarly, the detection window is shifted to the right edge of theU-disparity map, namely, the right stop position of such row is theright edge of the U-disparity map, as illustrated in FIG. 16.

Next, the detection window is shifted upward by one row, namely adistance equal to the height of the detection window, and repeating theabove operations of shifting the detection window to the left and rightand the left stop position and the right stop position of such a row isrecorded. It should be noted that the initial position of the row forshifting the detection window may be a corresponding position where theinitial position of the first row moving upward, or another positionreselected from the row within the drivable region.

After finishing the shifting of such a row, an operations of shiftingthe detection window upward by one row, shifting the detection window tothe left and right, and recording the left stop position and the rightstop position of the row is repeated until the detection window reachesa top edge of the U-disparity map or the left stop position and theright stop position in the same row are adjacent to each other.

Accompanying the shifting upward of the detection window, the conditionfor stopping the moving of the detection window may change. For example,FIGS. 17 to 21 illustrate the situation of shifting the detection windowto the left and right in the middle row. In this case, the moving of thedetection window is stopped because the number of the active pointswithin the detection window is greater than the number threshold, whenthe detection window is shifted to the left and right.

Next, a region covered by the shifted detection windows is extractedfrom the U-disparity map as the drivable region, as illustrated in FIG.22. In this way, the initial drivable region can be gradually extendedand the final drivable region can be obtained by shifting the detectionwindow row by row in the U-disparity map. It should be noted that themethod of shifting the detection window row by row as described above isjust an example; the present invention is not limited to this method,and other shifting methods, such as shifting the detection window columnby column may also be applied, as long as the above specific region canbe finally detected.

Returning to FIG. 4, in step S405, the drivable region detected from theU-disparity map is converted into the drivable region within thegray-scale map.

Specifically, coordinates of points within the drivable region detectedfrom the U-disparity map (u′, d′) may be obtained, after the drivableregion is detected from the U-disparity map.

Because the expression of the road line v=kd+b has been determined inthe V-disparity map, v′ coordinates of points (v′=kd′+b) may becalculated by the expression based on the coordinates of the points (u′,d′), and the coordinates are the coordinates of the points within thegray scale image (x′=u′, y′=v′). FIG. 23 illustrates an example of thecoordinate conversion. As illustrated in FIG. 23, with respect to apoint PP (400, 800) within the U-disparity map, the v′ coordinate ofsuch a point can be calculated as 633, based on the expression of theroad line v=0.5d+233; therefore, the coordinate of such a point withinthe gray scale image is (400, 633).

Next, the drivable region within the gray-scale map is extracted basedon the coordinates of the points within the drivable region of thegray-scale map. FIG. 24 is a schematic drawing illustrating theextracted drivable region of the road in the above example.

From this, the above method according to the embodiment of the presentinvention has a wide application range because it does not depend on thewhite lines or a fence on the road. Additionally, in the above detectionmethod, the drivable region is detected by the U-disparity map, at thistime, the drivable region is detected by using the density of the activepoints within the detection window and considering the relationship ofthe stop positions of the adjacent rows; therefore, the effect of noisein the detection result may be reduced remarkably, so that this methodhas high anti-noise robustness and reliability.

In the following, the device for detecting the drivable region of theroad according to the embodiment of the present invention will bedescribed with reference to FIG. 25. Such a device can perform themethod described above.

As illustrated in FIG. 25, the detection device 250 comprises: a roaddetecting apparatus 251, a sub-disparity map generating apparatus 252, aU-disparity map generating apparatus 253, a U-disparity map detectingapparatus 254, and a drivable region converting apparatus 255.

The road detecting apparatus 251 is configured to derive a disparity mapfrom a gray-scale map including the road and detect the road from thedisparity map. FIG. 26 illustrates the detailed structure of the roaddetecting apparatus 251. As illustrated in FIG. 26, the road detectingapparatus 251 comprises: a V-disparity map generating unit 2511configured to convert the disparity map into a V-disparity map; and aroad line fitting unit 2512 configured to fit a road line representingthe road in the V-disparity map.

The sub-disparity map generating apparatus 252 is configured to remove apart with a height above the road greater than a predetermined heightthreshold from the disparity map so as to generate a sub-disparity map.The sub-disparity map generating apparatus 252 may generate thesub-disparity map according to the above method, and the descriptionthereof is omitted here.

The U-disparity map generating apparatus 253 is configured to convertthe sub-disparity map into a U-disparity map.

The U-disparity map detecting apparatus 254 is configured to detect thedrivable region from the U-disparity map. FIG. 27 illustrates thedetailed structure of the U-disparity map detecting apparatus 254. Asillustrated in FIG. 27, the U-disparity map detecting apparatus 254comprises: a selection unit 2541 configured to select an initialposition within the drivable region from the U-disparity map; and adetection unit 2542 configured to detect a specific region from theU-disparity map as the drivable region by shifting a detection windowwith a predetermined size from the initial position in the U-disparitymap, wherein the specific region includes the initial position, and thenumber of active points with a brightness greater than a brightnessthreshold within the detection window is less than a predeterminednumber threshold during the time when the detection window is shifted inthe specific region. As described above, the initial position ispreferably a middle position of the lowest end of the U-disparity map.Additionally, the detection unit 2542 may shift the detection window bythe above method (such as 8-CCA method) so as to detect the drivableregion from the U-disparity map.

The drivable region converting apparatus 255 is configured to convertthe drivable region detected from the U-disparity map into the drivableregion within the gray-scale map. FIG. 28 illustrates the detailedstructure of the drivable region converting apparatus illustrated 255.As illustrated in FIG. 28, the drivable region converting apparatus 255comprises: a coordinate extracting unit 2551 configured to obtaincoordinates of points within the drivable region detected from theU-disparity map; a conversion unit 2552 configured to calculatecoordinates of points within the drivable region of the gray-scale mapbased on the coordinates of the points, by an expression for fitting theroad line in the V-disparity map; and a drivable region extracting unit2553 configured to extract the drivable region within the gray-scale mapbased on the coordinates of the points within the drivable region of thegray-scale map.

The road detecting apparatus 251, the sub-disparity map generatingapparatus 252, the U-disparity map generating apparatus 253, theU-disparity map detecting apparatus 254, the drivable region convertingapparatus 255, and their components may perform the above operationillustrated in FIG. 4, and the description thereof is omitted here.

The basic principle of the present invention is described above withreference to the embodiments. Any one or all of the steps or units ofthe method or device according to the present invention may beimplemented by hardware, firmware, software or their combination in anyone of computing devices (including a processor, a storage medium, etc.)or a network of computing devices, and it can be implemented by personsskilled in the art who have read the specification of the presentapplication.

For example, FIG. 29 is schematic drawing illustrating the hardwareconfiguration for implementing the above device. As illustrated in FIG.29, the device may comprise: an input apparatus 290 for inputtingrelevant images or information such as left and right images picked upby a binocular camera or stereo video picked up by a stereoscopiccamera, for example, including a keyboard, a mouse, and a communicationnetwork and an input device connected thereto, etc.; a processingapparatus 291 for receiving the images from the input apparatus 290 andimplementing the above method for detecting the drivable region of theroad according to the embodiment of the present invention or beingimplemented as the above apparatus for detecting the continuous roadpartition with a height according to the embodiment of the presentinvention, such as a CPU of a computer or other chips having processingability (as illustrated in FIG. 3), etc.; a storage apparatus 292 forstoring the images received from the input apparatus 290 and variousdata and/or results generated in the operation process performed by theprocessing apparatus 291, etc., by a volatile method or a nonvolatilemethod, such as various kinds of volatile or nonvolatile memoryincluding a random-access memory (RAM), a read-only memory (ROM), a harddisk and a semiconductor memory; and an output apparatus 293 foroutputting the result obtained by implementing the detected drivableregion to the outside, such as a screen, a printer, a communicationnetwork and a remote output device connected thereto, etc.

The present invention can also be realized by a program or a set ofprograms running on any one of computing devices. The computing devicesmay be well known general-purpose devices. Therefore, the presentinvention may also be implemented by providing a program productincluding program codes for implementing the method or apparatus. Thatis to say, the program product also belongs to the present invention,and a storage medium storing the program product also belongs to thepresent invention. Obviously, the storage medium may be any one of wellknown storage media or storage media which are to be developed.

In addition, in the device or method of the present invention, units orsteps may be divided and/or recombined. The division and/orrecombination should be regarded as an equivalent embodiment of thepresent invention. Steps of the above method may be performed in timeorder, however the performing sequence is not limited to the time order.Any steps may be performed in parallel or independently.

Furthermore, the present invention is described under the situation ofthe vehicle driving assistance technology, however the present inventionmay also be applied to other fields such as vehicle detection, trackingand driving early warning.

The present invention is not limited to the specifically disclosedembodiments, and various modifications and replacements may be madewithout departing from the scope of the present invention.

The present application is based on and claims the benefit of priorityof Chinese Priority Application No. 201210358202.4 filed on Sep. 24,2012, the entire contents of which are hereby incorporated by reference.

What is claimed is:
 1. A method for detecting a drivable region of aroad, comprising the steps of: deriving a disparity map from agray-scale map including the road and detecting the road from thedisparity map; removing a part with a height above the road greater thana predetermined height threshold from the disparity map so as togenerate a sub-disparity map; converting the sub-disparity map into aU-disparity map; detecting the drivable region from the U-disparity map;and converting the drivable region detected from the U-disparity mapinto the drivable region within the gray-scale map.
 2. The method fordetecting a drivable region of a road according to claim 1, wherein thestep of detecting the road from the disparity map comprises the stepsof: converting the disparity map into a V-disparity map; and fitting aroad line representing the road in the V-disparity map.
 3. The methodfor detecting a drivable region of a road according to claim 2, whereinthe sub-disparity map is generated by removing a part that is locatedabove the road line and has a distance from the road line greater thanthe predetermined height threshold from the disparity map.
 4. The methodfor detecting a drivable region of a road according to claim 1, whereinthe step of detecting the drivable region from the U-disparity mapcomprises the steps of: selecting an initial position within thedrivable region from the U-disparity map; and detecting a specificregion from the U-disparity map as the drivable region by shifting adetection window with a predetermined size from the initial position inthe U-disparity map, wherein the specific region includes the initialposition, and the number of active points with a brightness greater thana brightness threshold within the detection window is less than apredetermined number threshold during the time when the detection windowis shifted in the specific region.
 5. The method for detecting adrivable region of a road according to claim 4, wherein the step ofdetecting a specific region from the U-disparity map as the drivableregion by shifting a detection window with a predetermined size from theinitial position in the U-disparity map comprises the steps of: (a)horizontally shifting the detection window along a direction of rows tothe left from the initial position, and stopping the shifting of thedetection window and recording the stop position of the detection windowas a left stop position of the said row if a first determined conditionis met, horizontally shifting the detection window along a direction ofrows to the right from the initial position, and stopping the shiftingof the detection window and recording the stop position of the detectionwindow as a right stop position of the said row if a second determinedcondition is met; (b) shifting the detection window upward by a distanceequal to the height of the detection window, and repeating step (a); (c)repeating step (b) until the detection window reaches a top edge of theU-disparity map or the left stop position and the right stop position inthe same row are adjacent to each other; and (d) extracting a regioncovered by the shifted detection windows from the U-disparity map as thedrivable region, wherein the first predetermined condition is one of thecondition that the number of the active points within the detectionwindow is greater than a predetermined number threshold; the conditionthat the detection window reaches a left edge of the U-disparity map,even though the number of the active points within the detection windowis not greater than the predetermined number threshold; and thecondition that the horizontal distance between the position of thedetection window and the left stop position of the previous row is lessthan a first distance threshold, even though the detection window doesnot reach the left edge of the U-disparity map and the number of theactive points within the detection window is not greater than thepredetermined number threshold, wherein the second predeterminedcondition is one of the condition that the number of the active pointswithin the detection window is greater than a predetermined numberthreshold; the condition that the detection window reaches a right edgeof the U-disparity map, even though the number of the active pointswithin the detection window is not greater than the predetermined numberthreshold; and the condition that the horizontal distance between theposition of the detection window and the right stop position of theprevious row is less than a second distance threshold, even though thedetection window does not reach the right edge of the U-disparity mapand the number of the active points within the detection window is notgreater than the predetermined number threshold.
 6. The method fordetecting a drivable region of a road according to claim 2, wherein thestep of converting the drivable region detected from the U-disparity mapinto the drivable region within the gray-scale map comprises the stepsof: obtaining coordinates of points within the drivable region detectedfrom the U-disparity map; calculating coordinates of points within thedrivable region of the gray-scale map based on the coordinates of thepoints, by an expression for fitting the road line in the V-disparitymap; and extracting the drivable region within the gray-scale map basedon the coordinates of the points within the drivable region of thegray-scale map.
 7. A device for detecting a drivable region of a road,comprising: a road detecting apparatus configured to derive a disparitymap from a gray-scale map including the road and detect the road fromthe disparity map; a sub-disparity map generating apparatus configuredto remove a part with a height above the road greater than apredetermined height threshold from the disparity map so as to generatea sub-disparity map; a U-disparity map generating apparatus configuredto convert the sub-disparity map into a U-disparity map; a U-disparitymap detecting apparatus configured to detect the drivable region fromthe U-disparity map; and a drivable region converting apparatusconfigured to convert the drivable region detected from the U-disparitymap into the drivable region within the gray-scale map.
 8. The devicefor detecting a drivable region of a road according to claim 7, whereinthe road detecting apparatus comprises: a V-disparity map generatingunit configured to convert the disparity map into a V-disparity map; anda road line fitting unit configured to fit a road line representing theroad in the V-disparity map.
 9. The device for detecting a drivableregion of a road according to claim 7, wherein the U-disparity mapdetecting apparatus comprises: a selection unit configured to select aninitial position within the drivable region from the U-disparity map;and a detection unit configured to detect a specific region from theU-disparity map as the drivable region by shifting a detection windowwith a predetermined size from the initial position in the U-disparitymap, wherein the specific region includes the initial position, and thenumber of active points with a brightness greater than a brightnessthreshold within the detection window is less than a predeterminednumber threshold during the time when the detection window is shifted inthe specific region.
 10. The device for detecting a drivable region of aroad according to claim 8, wherein the drivable region convertingapparatus comprises: a coordinate extracting unit configured to obtaincoordinates of points within the drivable region detected from theU-disparity map; a conversion unit configured to calculate coordinatesof points within the drivable region of the gray-scale map based on thecoordinates of the points, by an expression for fitting the road line inthe V-disparity map; and a drivable region extracting unit configured toextract the drivable region within the gray-scale map based on thecoordinates of the points within the drivable region of the gray-scalemap.